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1 HIERARCHICAL RESOURCE MANAGEMENT FOR A COMPUTING UTILITY 

2 CROSS REFERENCES 

3 The present application is cross-referenced to application entitled "Componentized Automatic 

4 Provisioning and Management of Computing Environments for Computing Utilities", having 

5 docket number YOR920030588PCT1 , and application entitled "Apparatus for Arbitration in a 

6 Computing Utility System", having docket number YOR920030589US1, ^ven dated herewith, 

7 and which are included herein by reference in entirety for all purposes. 

8 FIELD OF THE INVENTION 

9 The present invention is directed to provisioning and management of resources in a computing 

10 utility, which uses the resources to provide computing services to customers. It is more 

1 1 particularly directed to hierarchical resource management for the computing utility. 

12 BACKGROUND 

13 A hosted center provides computing services to multiple customers. Each customer is allocated a 

14 subset of the service provider's infrastructure resources, such as servers, to meet its needs. 

1 5 Customer needs change over time, and in particular, peak demands for resources can exceed 

16 average demands by orders of magnitude. A simple static allocation of resources to satisfy peak 

17 demand leaves resources underutilized. Reconfiguring infrastructure resources dynamically, in 

1 8 response to customer needs, demands prompt attention from administrative personnel and could 

19 require moving hardware, increasing both operational costs and the risk of failing to provide 
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1 adequate service. The problem for the hosted center is how to respond quickly to changes in 

2 customer needs such that infrastructure resources and staff are employed in an efficient and cost 

3 effective way. Computing utilities attempt to address this problem by automating the creation and 

4 management of multiple computing services on shared dynamically allocatable infrastructures. : 

♦ 

5 Previous work in computing utilities varies in the types of services that are offered, the resources 

6 that are used, and the extent to which operation is automated. Operations subject to automation 

7 are wide ranging, and include creating services, deploying a service for a customer, modifying 

8 the set of resources used to provide the service, and incorporating new resource instances and 

9 types into the hosted center and its services. 

10 A previous system automated the provisioning of front end servers in web sites based on metrics 

1 1 such as server load and response time. It included a component that discovered server and 

12 network topology automatically. Another system provided a variety of multi-tier web sites in 

13 which pre-configured servers could be allocated to different tiers automatically based on metrics 

1 4 such as server load. Yet another system also allocated server resources in response to server load, 

.* 

1 5 but modeled both the value of allocating resources to each customer and the cost of employing 

1 6 those resources, with an emphasis on energy cost. More recent work includes allocation of other 

1 7 resource types such as memory and storage, and allocation of servers for general use. 

18 At the application layer some systems have a framework for deployment and management of 

1 9 distributed applications. An application is described as a collection of related, reusable 

20 components, which may represent resources or subsystems. The description includes 

2 1 dependency information to ensure that, for example, components are started in the correct 

22 sequence. Once deployed, applications may be monitored, and actions may be specified in case 

23 of component or resource failures, such as automatic fail over or restart. Such a system is not 

24 used for low level resource configuration tasks such as installing operating systems on servers, 

25 but for higher level application specific configuration. 1 

i 

26 A growing number of industrial products aim to provide multi-tier applies iions over a physical 1 

]A 
4 
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1 infrastructure consisting of a variety of resources, such as those from Hewlett Packard, 

2 ThinkDynamics, Sun Microsystems, and Jareva. They vary in many respects, such as the types of 

3 resources provided (e.g., servers and storage); specific operating systems and middleware 

4 supported; assumptions and characteristics of the network infrastructure (e.g., whether or not 

5 network isolation is provided via VLAN); level of monitoring support (e.g., resource usage, 

6 failure detection, SLA, threshold based alerting); support for resource discovery; support for 

7 modifying service resources once deployed; whether modifications can occur automatically (e.g., 

8 triggered by SLAs); and the extent to which the products can or must be customized to fit 

9 preexisting hosted center infrastructures. 

# 

10 SUMMARY OF INVENTION 

1 1 This invention provides methods, apparatus, systems and architectures for a hierarchical 

12 provisioning and management of a computing infrastructure, used to provides service oriented, 

13 or utility computing. It is useful for hosted environments. It is often provided to customers of a 

14 service provider who owns an infrastructure. It allows the dynamic provisioning and assignment 

15 of resources to computing environments. 

16 An aspect of the present invention is to provide an element of a computing utility that operates 

17 in an environment that possesses the following characteristics. Together, these characteristics 

1 8 generalize the environments studied in previous work. 

1 9 Another aspect of the present invention is to provide for automatically provisioning and 

20 managing resources for a computing utility. A computing utility could be used by a corporation, 

21 a service provider, or an individual.. This invention allows the sharing of resource within sets of 

22 customers and allows for restricting resources to specific customers. It allows customers to offer 

23 one or more services composed of managed resources to their clients. Managed resources may be 

24 assigned to customers or domains on demand. It also allows resources to be hierarchically 

25 managed. 
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1 In an example embodiment of a method according to the present invention, the method 

2 comprises the step of providing hierarchical management of at least one domain for an entity. ; 

3 The step of providing hierarchical management including: obtaining a hierarchical representation 

■ 

4 of the at least one domain, the representation including: a list of computing environments to be 

5 managed, at least one policy controlling acquisition of at least one resource from resource 

6 libraries for the at least one domain, and any sub-domains within the at least one domain; and 

7 instantiating the representation. 

8 In an example embodiment of an architecture according to the present invention, the architecture 

9 for a computing utility comprises an apparatus to provide at least one service for a plurality of 

10 clients. The apparatus comprises: a Base Resource Distribution Service to allocate resources to 

1 1 the at least one service; the Base Resource Distribution Service having at least one collector; at 

12 least one Provisioned and Managed Resource Service coupled to the Base Resource Distribution 

13 Service to provision and manage the resources for the at least one service; and at least one Base 

.» 

14 Resource Library Service coupled to the Base Resource Distribution Service to provide 

1 5 reservation and allocation of resources. 

i 

16 BRIEF DESCRIPTION OF THE DRAWINGS i 

1 7 These, and further, aspects, advantages, and features of the invention will be more apparent from 

1 8 the following detailed description of a preferred embodiment and the appended drawings ' 

19 wherein: 

a 

"I 

20 Fig. 1 shows the components of a Hosted Environment; 

2 1 Fig. 1 A shows an example structure of a company; ^ 

22 Fig. 2 shows the components in a computing utility management system in accordance with J 

23 the present invention; 

A 

24 Fig. 3 shows the operations of a library component in a computing utility management system - 

•"i 
"j 
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1 in accordance with the present invention; 

2 Fig. 4 shows a the hierarchy of collectors within a computing utility in accordance with the 

3 present invention; 

4 Fig. 4 A shows applications allocated to a unit of a company in accordance with the present 

5 invention; 

6 Fig. 4 B shows how the applications of Fig. 4A can be structured for provisioning in 

7 accordance with the present invention; 

8 Fig. 5 shows a computing utility with multiple root collectors in accordance with the present 

9 invention; 

1 0 Fig. 6 shows the association of resource pools to collections in accordance with the present 

1 1 invention; 

12 Fig. 7 show how acquisition policy is checked within a computing utility in accordance with 

1 3 the present invention; 

14 Fig. 8 show how to locate available resources within a computing utility; 

1 5 Fig. 9 shows the interaction between a Provisioned and Managed Resource Service (PMRS) 

16 and a Base Distribution Resource Service (BRDS) in a computing utility in accordance with 

1 7 the present invention; 

1 8 Fig. 1 0 show how a collector functions during the operation of a computing utility in 

1 9 accordance with the present invention; 

20 Fig. 1 1 Illustrates how a new service is added to a computing utility in accordance with the 

2 1 present invention; 

22 Fig. 12 Shows applications assignment within a company in accordance with the present 

23 invention; and 

24 Fig. 13 shows the collector hierarchy associated with the assignment of Fig. 12, in accordance 

25 with the present invention. 

26 DETAILED DESCRIPTION OF THE INVENTION 

27 This invention provides methods, apparatus, systems and architectures for hierarchical 
YOR920030587PCT1 5 



provisioning and management of a computing infrastructure which is used to provides service 



2 oriented, or utility computing, to the customers of the service provider who owns the 

3 infrastructure. In this way hosted environments can acquire resources from other hosted 

4 environments, etc. The invention provides an architecture for hierarchical management of 

5 computing infrastructures which included arbitration, provisioning and management. It allows 

6 the dynamic provisioning and assignment of resources to computing environments. Customers 

7 can have multiple computing environments within their domain. The computing utility shares its 

8 resources across multiple customer domains and arbitrates on the use of resources between and 

9 within domains. The invention enables resources to be dedicated to a specific customer domain 

10 or to a specific computing environment. Customers can specific acquisition and distribution 

1 1 policy which controls their use of resources from the utility within their domains. 

12 The present invention is an element of a computing utility that generally, but not necessarily, 

13 operates in an environment that possesses one or more of the following characteristics. Together, 

14 these characteristics generalize environments studied in previous work. 



15 



First, resources may be allocated to customers in combinations which are heterogeneous, 
may be interdependent, and vary over time. 



16 



17 



Second, the services provided to each customer may be different. For example, one 
customer may be provided resources for a web site, and another for a scientific computing 
cluster. Resource types, quantities, dependencies, and allocation patterns will thus vary 
between customers. 



18 



19 



20 



21 



Third, the level of service provided to each customer may be different. This means that 
evaluating the quality of a resource allocation to a customer considers both the type and 
level of service. 



22 



23 



24 



Fourth, the resource infrastructure vanes between service providers. Further, for a given 
service provider, the infrastructure varies over time. These variations can be a result of 



25 
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upgrades or additions to the physical infrastructure, engaging the services of another 
provider, or inclusion of additional resources when idle or at certain times of day, or 
removal of resources which are no longer needed. 

Fifth, resources may be allocated in advance, or reserved. A reservation may be accepted 
even if there are not sufficient resources currently available in the infrastructure to fulfill 
it, based on the expectation that additional resources will be acquired in time for 
allocation. 

Sixth, customers may want to share resources within an organization by subdividing their 
allocations and having the service provider manage the subdivisions, such as to 
departments. 

Seventh, customers may want to supply resources which they currently own to be 
managed by the service provider (i.e., the customer outsources the management of its 
resources to the service provider), which can only be used for their purposes. This 
requirement imposes constraints on where resources may be allocated. 

Eighth, customers may wish to specify policies which govern the management and 
operation of their resources. These policies are to be implemented by the computing 
utility. 

Finally, different service providers may have different objectives in mind when allocating 
resources to customers, such as maximizing profit, availability, or performance. 

This invention is also useful as a part of an apparatus for automatically provisioning and 
managing resources for a computing utility. A computing utility could be used by a corporation, a 
service provider, or an individual.. This invention allows the sharing of resource within sets of 
customers and allows for restricting resources to specific customers. It allows customers to offer 
one or more services composed of managed resources to their clients. Managed resources may be 
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1 assigned to customers or domains on demand. It also allows resources to be hierarchically 

2 managed. 

3 Figure 1 illustrates a Hosted Environment where the present invention runs. This environment is 

4 made up of processors 101, storage 103, firewalls 105, and software 107. The software 107 can 

5 be operating systems, middleware or applications. In Figure 1 the software 1 07 is represented as 

6 Business Processes, Business Applications, or Services. All of the available software has been 

7 pre-configured into these elements. In fact it is the purpose of this invention to enable the 

8 dynamic reconfiguration of the hardware and software components into new or existing services 

9 as needed. This type of environment can exist within a large company or it can be offered by an 

10 ISP or ASP as a basis for service. Returning to the detailed description, the hardware resources 

1 1 are connected by a network as indicated by the grid of lines 1 09 interconnecting all of these 

12 resources. This network may be configure into one or more tiers, where each tier is separated by 

13 a router 105 or firewall 105. The configuration of the tiers within the network may be static or 

14 dynamic. When a network is statically configured it is not possible without human (or 

1 5 mechanical) intervention to move a resource between tiers in the network. With a dynamic 

1 6 configuration, resources can be moved between different tiers under the control of an 

1 7 infrastructure, such as the one being described in this invention, or an operator working at a 

1 8 console. Within each tier Resources can be arbitrarily grouped into units. This can be done by 

19 using mechanisms such as Virtual LANs. Software resources are assigned to physical resources 

20 by the controlling infrastructure. 

21 In this environment a subset of the resources are assigned to the management infrastructure 111, 

22 1 13, & 1 15. In Figure 1 these resources are indicated by a dotted line surrounding them. The 

23 resources assigned to the management infrastructure run the Resource Management Software 

24 described in this invention. This software manages the rest of the resources. The resources used : 

25 for the management infrastructure are not assigned to clients 1 1 7 of the hosted environment. The 

* 

26 managed resources are assigned to clients 1 1 7 as necessary. It is expected that clients 1 1 7 will 

27 primarily receive service by connecting to the hosted environment thought the Internet 1 1 9. 

28 However, clients 1 1 7 can receive service if they are connected by any means to the hosted 
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1 environment. For example, they could have a direct connection to a managed resource, they 

2 could be connected to the same network as the managed resources, they could have a VPN 

3 connection to the managed resource, or they could have a VPN connection to the network 

4 containing the managed resources. Clients 117 have no view to the infrastructure; they only see 

5 the applications or services that they are receiving from the hosted environment. 

6 The management infrastructure assigns resources to each supported service as needed. The need 

7 is determined by the demand on the service in accordance with SLAs, contract, or other 

8 agreement that authorized the service to operate in the hosted environment. Individuals or 

9 organizations that wish to offer services will have agreements with the provider of the hosted 

10 environment. Returning to Figure 1, the processors 101 may have direct attached storage that 

1 1 would be managed as part of managing the processor. Storage 1 03 in this Figure refers to any 

12 form of storage server that can be network attached. The processors can also represent some 

1 3 complex function like a database server or a video server. The degree to which a resources is 

14 divided into smaller functions for management is dependent on the owner of the infrastructure. 

1 5 Even if a complex resource such as a database server can be broken down into smaller 

16 components, the owner of the resource will determine how it is represented within the 

1 7 management infrastructure. This feature allows the invention to use resources that are not local to 

1 8 the hosted environment. The invention assumes that the resources are securely interconnected 

1 9 with adequate performance. However, the network which connects the resource represents a 

20 logical connection. Any infrastructure, such as a VPN, which allows an adequate degree of 

21 security, performance, and control over a resource, can be used to connect resources into the 

22 hosted environment. Consequently the hosted environment as depicted in Figure 1 could itself be 

23 distributed world wide. The Internet in this picture represents a logical placement the Internet 

24 can be used to dynamically add resources to the hosted environment. 

25 A domain is an organizational unit within a customer for which services are provisioned. An 

26 organization may have multiple sub domains, division, units, or departments within it which also 

27 utilize the service provider. A sub domain is also a domain and the domains of an organization 

28 are form a tree. An organization can have multiple trees of domains which are not connected. For 
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the purposes of this invention, we consider those to be multiple organizations. The root domain 
is the highest domain or the base of the domain tree for an organization. Figure 1 A shows the 
domains within a customer where services can be provided. In Figure 1 A the main company, or 
root domain, is Smith Fastener Company 120, all other parts of this company are subdivisions, 
or sub-domains, of Smith Fastener. Smith Fastener has four main divisions, Research 121, 
Finance 122, Hardware 123, and Marketing 124. The hardware and marketing division have 
additional units, or sub domains, within each of them. Hardware is divided into Bolts 127, screws 
128, and hinges 129. Marketing has two regions Region 1, 125 and Region 2, 126. Region 2 is 
further divided into North 130 and South 131. It is an object of the present invention to allow 
service providers to provide computing services to the various domains, units, and/or division of 
any company, such as the Smith Fastener company. Any domain or sub-domain may then 
subscribe to the computing services provided. This invention assumes that for provisioning 
purposes the company can be structured into a domain tree. Further any ai oitrary tree can be 
supported. As will be described later, the domain tree controls the sharing of resources for the 
purposes of provisioning between the various applications that are used by a company. This 
invention does not require that the resource sharing tree exactly map to the corporate structure. 
Any arbitrary resource sharing tree can be used. For illustrative purposes we will use one that 
maps to the structure of an organization, Smith Fastener Company 120. 

Within an organization, each domain, division, or unit has its own employees. This invention 
does not restrict who (which employees of the corporation) within a corporation can use the 
computing services. Eligibility to use a computing environment is controlled by the 
administrative policies within the corporation. User Ids, passwords, and authorization are 
controlled by the corporation. For example, all of the application could use a common 
authentication and authorization mechanisms, each application could have its own, or any 
combination in between. The provisioning of access control is part of the PMRS. The use, (user 
who are allowed to access or use the applications) of computing services is independent of the 
structure used to provision resources to the applications. 

In this environment there are multiple ways to divide the roles and responsibilities for utilizing 
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■1 

1 this invention. In one such division there is an infrastructure provider who owns the hosted 

2 environment described in Figure 1 ; one or more Service Providers offering some service on ■ 

3 using the hosted environment; and one or more clients of the Service Provider. Note that the j 

4 service provider is also a customer of the infrastructure provider. In another such division a 

5 company (or entity) may contract with an infrastructure provider for services, outsourcing the : 

6 management of its infrastructure. In this case the company is utilizing this invention to manage 

7 the services provided within itself. In another such division the infrastructure provider and 

8 service provider could be the same. Further there are entities that develop the services that are 

9 offered. They could be separate, such as an independent software vendor (IS V), or part of one or 

10 more of the previously named entities. An entity who wants to offer a service that is managed by 

1 1 this invention on some infrastructure provides a description of that service to the infrastructure. 

12 This description can be abstract or concrete and includes the organization of the service, an 

13 indication of the service to be provided and the policies which control the operation of the 

14 service. 

1 5 Figure 2 shows a high level view of a computing utility. There are three layers of components. 

1 6 The top layer represents the services which are provisioned, called computing environments. A 

.♦ 

1 7 computing environment includes hardware and software resources such a? servers, operating 

1 8 systems, applications, and middleware. Each computing environment has a service which is 

1 9 invoked to provision and manage its resources called the Provisioned and Managed Resource 

20 Service (PMRS). The PMRS is described in detail in a separate invention entitled 

2 1 "Componentized Automatic Provisioning and Management of Computing Environments for 

22 Computing Utilities", having docket number YOR920030588PCT1. The bottom layer 

23 represents the resources of the hosted center available to its customers. A base resource is a 

24 resource that is atomic; it is not broken down into other resources. Each base resource type has a 

25 service that provides reservation and allocation of instances of that resource type, called the Base 

26 Resource Library Service (BRLS). The definition of base resource is up to the service provider. 

27 Therefore, a BRLS can provide a simple resource such as a software license or a complex 

28 resource such as an operating system installed and running on a hardware platform. Adding, 

29 removing, or modifying a base resource changes the overall capacity of the system. This 



YOR920030587PCT1 



1 invention assumes that both the number and types of the base resources change over time. Base 

2 resources are represented by a set of Base Resource Library Services (BRLS) 201. Figure 2 

3 shows four types of base resources: DB2 license 203, zSeries LPAR 205, xSeries Server 207, and 

4 ADC license 209. The BRLS provides operations such as cataloging, check out (allocation), check 

5 in (deallocation), and reservation. The interfaces to a BRLS 203, 205, 207, 209 are described in 

6 more detail in Figure 3. 

7 A composite resource is built out of one or more other resources (including other composite 

8 resources) to perform a specified function. A composite resource may have a set of 

9 dependencies associated with it. An example of a composite resource is a web site. It may be 

1 0 composed of some number of front-end servers, back end servers, a load balancer, a set of IP 

1 1 addresses for the servers, web server software, database software, and licenses associated with 

1 2 the software. The set of base resources used to implement the function of a composite resource 

1 3 can change over time, though not all composite resources may have this capability. The 

14 computing environment is a type of composite resource. 

1 5 Associated with each computing environment is software that provisions and manages the 

16 resources used to provide the service, called the Provisioned and Managed Resource Services 

1 7 (PMRS) 211. Every resource type, including composite resources, has a PMRS which 

1 8 encapsulates the knowledge of how to create a resource of that type, and how to manage an 

19 instance of the resource. Provisioning refers to the act of allocating a resource to a computing 

20 environment and configuring it for use within the service. Provisioning actions may include 

21 assembling base resources into composites, configuring network devices, installing operating 

22 systems, application software, monitors, and user accounts. Managing the resource may include 

23 such actions as requests for adding or removing capacity from the resource instance. Figure 2 

24 shows the PRMSs, DB2 213, Linux 2 1 5, and Web Site 2 1 7. The PMRS is described in detail in 

25 invention entitled "Componentized Automatic Provisioning and Management of Computing 

26 Environments for Computing Utilities", having docket number YOR920030588PCT1 . 

27 The middle layer of Figure 2 is the Base Resource Distribution Service (BRDS) 219. Its job is 
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to allocate the resources of the service provider to its domains in an effective way, based on the 
agreements that the service provider has in force for each of its domains. Domains are 
hierarchically organized, some domains are sub domains of others. The BRDS specifies which 
sets of resources are available to which domains and automatically redistributes those resources 
based on business policies when resources are constrained It interacts with both the computing 
environments 21 1 (PMRSs) and resources 201 (BRLSs). A resource may be allocated to a 
computing environment either immediately or in the future. The promise of a future resource 
allocation is a reservation. This invention assumes that the resources allocated to a computing 
environment are dedicated to that computing environment for the duration of their allocation. 

Base Resource Library Service 

Resources reside in pools, which may be arranged by type. Resource pools may be public or 
private. A public resource pool is a resource pool from which any domain may allocate a 
resource. A private resource pool is restricted to a subset of customer domains. Private pools 
are used to dedicate resources to a subset of customer domains. For example, a private pool could 
hold resources for use in multiple domains on behalf of a single customer. Such a pool might 
also be used for resources the customer owns but wishes the service provider to manage for it. 
Every resource has a home pool from which it originates, which may be either public or private. 
The home pool does not change when a resource is allocated or reserved. A group of unallocated 
resources is called a free pool. 

Figure 3 shows an example of an operation of a Base Resource Library Service 310 (BRLS). 
Every base resource in the system is represented by a library, or BRLS 310, which provides 
information such as cataloging, check out (allocation), check in (deallocation), and advance 
reservation. A BRLS serves as a home pool for instances of a resource type, and it may be either 
public or private (represent a public resource pool or a private resource pool). Each BRLS 3 10 is 
associated with a collector. Every BRLS 310 has two types of interfaces, Resource Operations 
330 which are used by the system to, acquire, reserve, and return resources and Catalogue 
operations 340 which are used manage the resources available to a BRDS 219. The Resource 
Operations 330 provided are Reserve, CancelReservation, Checkln, Checkout, 
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1 ExtendReservation, and Query. 



2 Reserve (num-instances, selection- spec, start time 

3 -> 

4 This request is issued by BRDS 219 on behalf of a customer computing 

5 environment to reserve resources. 

6 Inputs: 

7 Num-instances is the number of instances desired 

8 Selection-spec is a specification of desired attributes supported by the 

9 BRLS310. 

1 0 Start time, If unspecified, any instance may be used to satisfy the request. 

1 1 Start time could be immediate or for an advance reservation 

12 Outputs: 

13 Reservation-ids[]. The elements are resource reservation tickets, one per 

14 resource instance, up to num-instances. These need not map to instances 

1 5 at the time of the reservation. In other words, the library may overbook its 

16 resources. 



17 Checkout (reservation-id) -> resource-handle 

18 This request is issued by BRDS 2 1 9 on behalf of a customer computing 

1 9 environment to allocate resources. 

20 Inputs: 

2 1 Reservation-id issued by Reserve 

♦ 

22 Outputs: 

23 Resource-handle. An identifier of a resource instance. Set to a special 

24 value if an instance of the resource cannot be provided. 

25 Checkln (resource-handle) 

26 This request is issued by BRDS 219 on behalf of a customer computing 

27 environment to return resources. 
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1 Inputs: 

2 Resource-handle issued by Checkout. 

3 ExtendReservation (resource-handle, end time) 

4 This request is issued by BRDS 219 on behalf of a customer computing 

5 environment to extend the reservation for a currently held resource. 

6 Inputs: 

7 Resource-handle issued by Checkout. 

8 End time: the point at which the resource will be returned. 

9 Outputs: 

1 0 Accept: Resource reservation can be extended. 

1 1 Reject: Resource is returned. 

12 Query (selection-spec) -> availability-data 

1 3 This request is issued by BRDS 2 1 9 on behalf of a customer computing 

1 4 environment to return resources. 

15 Inputs: 

16 Selection-spec is a specification of desired attributes supported by the 

17 BRLS310. 

18 It is used to restrict the scope of the query. 

1 9 It may contain a resource handle to refer to a specific instance. 

20 Outputs: 

21 Availability-data[], The elements are structures that indicate when 

22 instances are available for reservation. 



23 CancelReservation (reservation-id) 

24 This request is issued by BRDS 219 on behalf of a customer computing 

25 environment to cancel a reservation. After a cancellation is received, the resource is 

26 available to be assigned and checked out by another computing environment. 

27 Inputs: 



YOR920030587PCT1 



15 



Reservation-id issued by Reserve 

Catalog Operations 

Catalog Operations 340 are provided for administrators 320 or other administrative mechanisms 
to modify the resource pools managed by the BRLS 310. Resource discovery mechanisms could 
also use these operations to automate this process. The operations provided include Add, 
Remove, Update, Query, and any combination of these operations. 

■ 

Add (resource-identifier , instance-data) 

This request is issued to add a resource instance to the BRLS 310. 
Inputs: 

Resource-identifier is a resource identifier, may or may not be the same as 
the handle above. 

Instance-data is the collection of attribute values that can be used to select 
the resource via a selection-spec 

Remove (resource-identifier) 

This request is issued to remove a resource instance from the BRLS 3 1 0. 
Inputs: 

Resource-identifier is the resource identifier 

Query (selection- spec) -> resource-identif ier [] 

This request is used to look up a resource instance registered with the BRLS 310. 
Inputs: 

Selection-spec is a specification of desired attributes supported by the 
BRLS 310. 

It is used to restrict the scope of the query. 

It may include a resource identifier. 

Outputs: 
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1 Resource-identifier[] a list of resource identifiers that satisfy the 

2 selection-spec 
3 

4 Query (resource-identifier) -> instance-data 

5 This request is used to look up a resource instance registered with the BRLS 310. 

6 Inputs: 

7 Resource-identifier is the resource identifier 

8 Outputs: 

9 Instance-data is the collection of attribute values registered for this 
1 0 instance, if any 

11 

12 Update (resource-identifier, update-data) -> instance-data 

1 3 This request is used to modify a resource instance registered with the BRLS 3 1 0. 

14 Inputs: 

1 5 Resource-identifier is the resource identifier 

1 6 Update-data new information to be applied to or registered about the 

1 7 resource 

18 Outputs: 

1 9 Instance-data is the collection of attribute values registered for this 

20 instance, if any 

2 1 Base Resource Distribution Service 

22 Figure 4 shows an example of components of a Base Resource Distribution service (BRDS) 219. 

23 The BRDS 219 determines how to distribute the base resources across the various computing 

24 environments. The BRDS 219 includes of two types of components, Collectors 420, 422, 424, 

25 426, 428 and an Arbiter 430. A domain is associated with at least one collector. For every 

26 computing environment there is one collector which includes the policies previously mentioned, 

27 and the list or resources which are reserved for the computing environment. 
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Arbiter 



The arbiter determines how resources are divided amongst the computing environments. It 
operates both on current and future allocations (reservations). Resources may be allocated to 
computing environments either reactively or proactively. The arbiter can be consulted when free 
resources are not sufficient to fulfill demands from customer computing environments. It can 
also be used on a periodic basis to optimize resource allocations. If resources are constrained, 
the arbiter may reclaim resources from computing environments. Details .on the operation of the 
arbiter are in a separate invention entitled "Apparatus for Arbitration in a Computing Utility 
System", having docket number YOR920030589US1. 

Collector 

A collector represents the set of resources allocated to one or more computing environments. 
The root domain of each computing environment has an associated collector. Collectors may be 
nested, for example, to represent departments within an organization. This structure enables 
resource sharing between a subset of computing environments, essentially replicating the 
computing utility model for those domains. A computing utility can provision resources for 
multiple organizations, a collector that is connected to multiple organizations (the root domain of 
each organization) is called the root collector. Figure 4 shows one root collector 420 with two 
domains, one of which has two sub domains. The two main domains are Col-A 424 and Col-B 
422 representing Company A and Company B. Col-A 424 has two sub domains Col-X 426 and 
Col-Y 428 representing Departments X & Y. There are three PRMSs shown in Figure 4 
PMRS-X 496, PRMS-Y 498, and PRMS-B 495. A collector is associated with each of these 
computing environments. In addition the BRDS 219 also has a collector 420. 

Figure four also shows public and private resource pools. Public BRLSs are associated with a 
root collector in a BRDS 219. A private BRLS is associated with a collector other than a root 
collector of a BRDS 219. In Figure 4, BRLSs 486, 488, and 482 are public because they are 
associated with the root collector 420 and BRLS 484 is private because it is associated with 
collector 424, which is not the root collector 420. 
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The collector implements the customer's acquisition policy, which determines under which 
circumstances to add or remove resources from its collection, or change the composition of a 
collection. The hierarchical structure of collector allows local (organization specific) policy 
governing resources to be specified. The simplest acquisition policy (i.e., no acquisition policy) 
is entirely request driven. That is, a resource moves between BRLS 482 (486, 488) and PMRS 
495 through the collector hierarchy on every request and return. A more complex acquisition 
policy might specify minimum and maximum numbers of servers at each collector. These 
acquisition policies filter the request flow. Such an acquisition policy allows collectors to 
maintain a variety of free pools (caches of allocated or reserved resources) for its domain, in 
effect pre-allocating resources for predicted requests. Acquisition policy will be discussed in the 
description of figures 7 & 8. 

Each BRLS 482, 484, 486, 488 is associated with a collector. This association defines the scope 
of sharing for the resource pool represented by the BRLS. For example, if an organization wishes 
to have a private resource pool for its departments, it would have a BRLS 484 associated with 
the collector 424 of the organization as illustrated in Figure 4. The resources would be available 
for allocation only by customer computing environments associated with the organization (i.e., 
associated with a collector that has the organization's collector as an ancestor). 
Figure 4 A shows a sub domain, Hardware 123, of the Smith Fastener company from Figure 1 A. 
For provisioning purposes Hardware has been assigned three applications, Appl 440, App2 450, 
and App3 460. In this case all three application are allowed to share the same set of resources. 
Figure 4B shows how this decision on provisioning maps to a collector hierarchy. Col 1 470 is 
associated with Hardware 123. In addition each application has a PMRS and a collector 
associated with it. Appl 440 has Col 1 441 and PMRS 1 442, App 2 has Col 2 451 and PMRS 2 
452, and App3 has col 3 461 and PMRS 3 462. In this invention a PMRS is associated with at 

■ 

most one collector. 

Figure 5 shows an example of a computing utility with multiple root collectors 510 and 520. This 
invention describes a BRDS 219 with a single root collector. Those skilled in the art can extend 
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1 this invention to multiple roots by logically partitioning the resource pools and associating a 

2 BRLS for each partition with distinct root collectors. The BRLSs for the root collector could 

3 manage the same resource types. Figure 5 illustrates this concept by showing a BRDS 219 with 

4 two root collectors 510 and 520. Root collector 510 has BRLSs 530 and 535. Root collector 520 

5 has BRLSs 540 and 545. In this case BRLS 530 could be managing the same type of resources ad 

6 BRLS 540. Note that in an alternative implementation, BRLSs of the same type but different 

7 association can be represented as a single BRLS with constraints on where their resource 

8 instances may be assigned. 

9 Figure 6 illustrates an example of a collections and libraries. Each computing environment has a 

10 set of resources allocated to it. These are shown only in their corresponding PMRS 660, 680, 

1 1 and 690, not in collectors or their ancestors. Collector Col-Y 632 has an extra resource of type 

12 "triangle" allocated for use by computing environment Y. The home collection of the triangle is 

13 the private library BRLS-1 690. Triangles can only be allocated to PMRS X 660 and PMRS Y 

14 680. Collector Col-A 630 has extra resources of type "square", "star", and "circle" for use by 

1 5 computing environments Col-X 633 or Col-Y 632. They are each from a public BRLS of the 

16 corresponding type 692, 694, and 696 respectively. These resource instances could be reclaimed 

17 for use by collector Col-B 620. BRLS 690 in Figure 6 is private. Its resources are shared by 

1 8 collectors 630, 632, and 633. In Figure 6 BRLSs 692, 694, and 696 are public. 

* 

1 9 When a resource is requested by a computing environment, the BRDS 2 1 9 goes through a three 

20 stage process. The first stage is determining whether or not the acquisition policies of the 

2 1 collector for the computing environment and its intermediate ancestor nodes, if any, would be 

22 satisfied if the request is granted. The BRDS 2 1 9 may deny the request if for example the 

23 resources requested would exceed that specified in the customer's agreement with the service 

24 provider. Thus the acquisition policy is a set of constraints that is taken into account in this 

25 decision process. If the BRDS 219 determines that a resource can be granted, the second stage of 

26 the process is to locate the instances which will satisfy the request. Available instances may be 

27 found in a public or private BRLS, or in a collector. If no instances are available, then an 

28 instance may be reassigned from another computing environment according to a distribution 
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1 policy. If the request is for a set of resources, the BRDS 219 coordinates their allocation 

2 accordingly. If the request succeeds, a set of reservation tickets is returned to the PMRS. 

3 The BRDS 219 will attempt to satisfy the acquisition policies of collectors, even in the absence 

4 of resource requests from the PMRS, generating requests to a BRLS if necessary. It is the BRDS 

5 219 that makes reservations on behalf of a collector, whether it is provoked by acquisition policy 

6 or explicit resource requests. 

7 Returning to the description of the collector and BRDS 219, when the start time of a reservation 

8 arrives, the BRDS 2 1 9 transforms the ticket into a resource handle. If the transformation of the 

9 reservation into a resource handle fails, the reservation cannot be honored. If the collector 

10 holding the ticket is associated with a PMRS, the BRDS 219 delivers the handle to the collector 

1 1 who then presents it to the PMRS. In Figure 6 if collector 632 was holding a reservation ticket 

12 that became due, the BRDS would transform it into a resource handle which would be returned 

1 3 to collector 632 who would then pass the resource handle onto PMRS 680. The PMRS 680 can 

14 then configure the resource into the computing environment. Otherwise the collector holding the 

15 reservation is not associated with a PMRS so the BRDS 219 delivers the resource handle to the 

16 collector. In Figure 6, this would occur if a reservation were held by collector 630. 

1 7 When a reservation expires, the BRDS 219 reclaims the resource handle associated with it, and 

1 8 returns it to its lender (home pool or intermediate collector). The invariant maintained is that a 

19 resource is checked out of a home pool or lent by an intermediate collector if and only if there is 

20 a reservation for it (and presumably, a customer is being billed for it). 

21 To extend a reservation, a PMRS 660 issues a request to the BRDS 219 w>th a selection 

22 specification for the resources it is already using. The BRDS 219, if acquisition policy and other 

23 commitments allow, will interact with the BRLS (692, 694, 696 or 690) to extend the 

24 reservation. That is, the PMRS 60 does not communicate with the BRLS directly. 

25 When a resource is returned by the PMRS 660, the acquisition policy of its associated collector 
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1 determines if it is returned to its parent. If the collector retains the resource, the client may 

2 continue to be billed for the resource. One advantage of retaining the resource is faster 

3 deployment on request. (For example, metering for the resource is already configured.) If it 

4 returns the resource, the acquisition policy of its parent is similarly checked. Resources are not 

5 retained at collectors associated with their home pool (BRLS); they are checked into their home : 

6 pool in that case. For example if a resource associated with BRLS 690 were returned to collector 

7 Col- A 630, it would be returned to the BRLS 690 instead of being held at the collector. 

8 Note that a collector that acts as a resource pool for a set of computing environments (i.e., an 

9 intermediate node in collector hierarchy) such as 630 in Figure 6, requires the same functionality 

10 as a BRLS to distribute its resources. It is essentially a library that lends borrowed resources. As 

1 1 such, its resource collections can also be represented as BRLSs associated with it. The difference 

12 between a collector and a private BRLS is that resources may be reclaimed from a collector for 

13 use by its parent or other parts of the collector hierarchy (where ever the home BRLS of the 

14 resource allows it). In contrast, the resources in a private BRLS can only be used by its associated 

1 5 collector and its descendants. A collector can potentially have its own collection of every 

16 resource type known to the system. Like a private BRLS, an implementation of the collector's 

1 7 collections can be achieved within a single set of BRLSs, with constraints on where their 

1 8 resource instances may be assigned, and additionally, the identity of the collector that holds the 

1 9 reservation ticket for the resource from its home pool. Once a resource has been assigned to a 

20 collector or a PMRS, its location within the hierarchy becomes an attribute of the resource. 

21 Figure 7 illustrates an example of a process of checking acquisition policies on resource requests, 

22 whether a request is generated by a PMRS or as a result of a collector's acquisition policy. The 

23 process starts in box 710. We first check 720 to see if the request is initiated by a PMRS. If it is 

24 730 we set the current collection to be the PMRS so that acquisition policy checking begins with 

25 the collector associated with that PMRS. Otherwise 740 we set the current collection to be the 

26 requesting collector's parent. This is because the request was initiated by a collector whose 

27 acquisition policy requires more resources. Next we check 750 to see if we are at a root collector. 

28 If the policies are satisfied all the way to the root collector, then the resources can be obtained, 
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and flow passes 780 to a procedure which attempts to locate available resources. Otherwise we 
check 760 see if the request would violate the acquisition policy of the current collection. If the 
policy would be violated then we check 765 to see if the requester is a PMRS. If it is the request 
is denied 795. Otherwise, it may be possible to satisfy the request using resources already 
allocated to the collector whose acquisition policy would be violated and its descendants, and 
arbitration 790 is called on the part of the sub tree rooted at the collector whose acquisition 
policy would have been violated. Returning to the check at 760, if the policy would not be 
violated at the current collection, then we set 770 current collection to the parent of the 
collection that was just examined and continue checking at 750. 

Figure 8 illustrates an example of a process of locating available resources. The process starts in 
810. First the starting point of the search is determined. If the request originated from a PMRS 
820, the starting collection for the search is the collector associated with the PMRS 827. 
Otherwise, it is the parent of the collector from which the request originated 825. Since a 
resource request may specify a set of resources of different types, a list of the descriptions of the 
desired resources is constructed 830 , one element per resource type per instance. Those skilled 
in the art realize that there are multiple ways to form this list, such as one element per resource 
type with multiple elements of the same type being tracked within the element. For each element 
in the list, three alternatives are checked for resources: a private BRLS associated with the 
collector, the collector's collection, or a public BRLS (when the search reaches the root). These 
checks are repeated at each collector from the starting collector to a root. This is done by 
removing 840 the first requested resource RR, from the list of required resources. Beginning in 
step 850 with the starting collection, we check 860 to see if there is a BRLS with an available 
resource of type RR. If not we then check 863, to see if the collector has a resource of type RR. 
If not we check 865, to see if we are at the root collection. If we are not at the root, then we move 
867 to the parent of the current collection and continue checking at 860. If we are at the root, 
then not all of the required resources are available and arbitration 870 will be used to determine 
if the resources can be made available. If check 860 or check 863 finds the resource, then 
information on the resource is added 880 to the reserve list. Next we check 890 to see if all 
requested resources have been located. If they have not been located then we get the top element 
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1 off of the list 840 and continue looking for resources. If all resources have been located, then 

2 reservation requests 895 are made for the located resources and the result of the reservation 

3 request is returned as the end of the process to locate available resources. 

* 

4 If the reservation fails, arbitration will be called with the root collector as' scope. Arbitration is 

5 the subject of a cross referenced application entitled "Apparatus for Arbitration in a Computing 

6 Utility System" having docket number YOR920030589US 1 , the present invention treats 

7 arbitration as a "black box". If arbitration succeeds, the resources are reserved and tickets are 

8 returned to the requester. Otherwise, the request is denied. A successful arbitration may require 

9 resources to be reclaimed from other domains. The constraints imposed by the collection 

1 0 managers, associated libraries, and acquisition policies are input to the arbiter. 

1 1 The BRDS provides the Resource Management operations listed below. 

1 2 Resource Management Operations 

1 3 Figure 9 illustrates an example of the interactions between the BRDS 2 1 9 and the PMRS 920. 

■ 

14 There is one set of operations 930 for the BRDS to make requests of the PMRS and another set 

15 of operations 940 for the PMRS to make request of the BRDS. Allocation of resources involves 

16 five operations: the PMRS can request, retain, or return resources 940, and the BRDS can 

1 7 reclaim or offer resources 930. When a PMRS 920 needs additional resources or has excess to 

1 8 return it uses the RequestResource and ReturnResource interfaces 940. When the PMRS wants to 

1 9 keep resources beyond the duration specified on the initial request, it uses RetainResource 940. 

20 When resources that were previously reserved become available the BRDS 2 1 9 uses the 

21 DeliverResource interface 930 to give the resources to the PMRS 920. The reclaim resource 

22 interface 930 is used by the BRDS whenever it needs to remove resources from a PMRS. For 

23 example, one case would be revoking previously allocated resources so that they can be 

24 reallocated somewhere else. The OfferResource interface 930 is used to make available 

25 additional resources that were not explicitly requested by the PMRS 920 but are allowed with the : 

26 policies of its computing environment. 
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1 RequestRe source (R, start-time, duration) 

2 This request is issued by the PMRS 920 to the BRDS 219 on behalf of a customer 

3 computing environment to request additional resources. The BRDS 219 can refuse the 

4 request. 

5 Inputs: 

6 R[], a vector of structures, one element for each resource type. Each 

7 structure contains the number of instances of the type and optional 

8 selection criteria. The selection criteria may include resource handles to 

9 specify particular instances. 

1 0 Start time could be immediate or for an advance reservation 

1 1 Outputs: 

1 2 Return code indicating success or failure 

13 R'[], an optional return. On failure, may be set to indicate resources 

14 available. The elements of R 5 are resource reservation tickets, one per 

1 5 resource instance, which the PMRS 920 may redeem to acquire the 

16 additional resources. 

1 7 Start-time', an optional return. On failure, may be set to indicate when 

18 resources are available. Applies to R' if set, R otherwise. 

19 ReturnResource (R) 

20 This request is issued by the PMRS 920 to the BRDS 2 1 9 on behalf of a customer : 

21 computing environment to return resources. The BRDS 219 accepts the resources. 

22 Inputs: 

23 R[] a list of resource identifiers (handles or reservation tickets) for the 

24 resources being returned. These resources could be currently allocated 

25 resources or reservations. 

26 Outputs: 

27 Return code indicating success or failure 
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1 Retain Resource (R, end time) 

2 This request is issued by the PMRS 920 to the BRDS 2 1 9 on behalf of a customer 

3 computing environment to retain resources (extend or modify the reservation of 

4 resources) for a different amount of time from the previously committed time. The BRDS 

5 219 may accept or reject the request. 

6 Inputs: 

7 R[] a list of resource identifiers (handles or reservation tickets) for the 

8 resources being retained. These resources could be currently allocated 

9 resources or reservations. 

10 End time: is the requested new end time for the resources in the list. If the 

1 1 end time is not in the future the request is rejected. 

12 Outputs: 

1 3 Return code indicating success or failure 



14 ReclaixnResource (R) 

15 This request is issued by the BRDS 219 to a PMRS 920 to revoke the allocation 

16 or reservation of a set of resources. The PMRS 920 relinquishes the resources. 

17 Inputs: 

1 8 R[], a vector of structures, one element for each resource type. Each 

1 9 structure contains the number of instances of the type and optional 

20 selection criteria. The selection criteria may include resource handles or 

21 reservation tickets to specify particular instances. The computing 

22 environment may select the resource instances to be reclaimed according 

23 to the criteria. 

24 Outputs: 

25 R'[] a list of resource identifiers (resource handles or reservation tickets) 

26 being returned. R' may be a superset of R which includes resources that 

27 would be rendered useless because of dependencies on reclaimed 

28 resources. 
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OfferResource (R, start-time, duration) 

This request is issued by the BRDS 219 to a PMRS 920 to signal the availability 
of additional resources for this computing environment. The PMRS 920 may accept or 
decline the offered resources. To accept the resources the PMRS 920 issues a 
RequestResource request. 

Inputs: 

R[] a vector containing the number of instances of each resource type 
available 

# 

Start time could be immediate or for an advance reservation 
Outputs: 

None 

OeliverResource (R) 

This request is issued by the BRDS 219 to a PMRS 920 to indicate that previously 
reserved resources have been allocated for the computing environment. 
Inputs: 

R[] a list of tuples, each consisting of a reservation-id and corresponding 
resource-handle. 
Outputs: 

None 

To acquire and distribute resources to customer computing environment, the BRDS 219 invokes 
the operations of the BRLS 910, 912, 914, 916 described earlier. 

Figure 10 illustrates an example of the interactions between the PMRS 1010 and the BRDS 219 
during normal operations. The PMRS in Figure 10 represents a group of Linux servers which 
may be of variable size. Some event causes the PMRS 1010 representing the Linux server to 
realize that it needs additional resources, such as an xSeries Linux server and an additional IP 
address. It requests the resources from the BRDS 219 using RequestResource 1011. The BRDS 
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219 uses the methods described in figures 7 and 8 to allocate and reserve the resources. The 
reservation tickets are returned to the PMRS 1010 in the response to Reqi ^stResource 1011. 
When the resources become available the PMRS 1010 will be informed using DeliverResource 
1012. The detailed interactions inside of the PMRS 1010 are described in an application entitled 
"Componentized Automatic Provisioning and Management of Computing Environments for 
Computing Utilities", having docket number YOR920030588PCT1. 

Figure 1 1 illustrates an example of a process of adding a new customer (or service) to an existing 
hosting infrastructure. This process is analogous to adding a new resource to an existing service. 
This invention assumes that there is a set of operations that allows organizations to subscribe to 
the services of the service provider, and further, to add computing environments in groups for 
resource sharing within an organization. At a minimum, to add an organization certain 
information from the agreement between the organization and service provider, such as the start 
time and duration of the agreement and acquisition or other polices is supplied. For specific 
computing environments, the computing environment type and settings arc also supplied. During 
normal operations the policies controlling a computing environment can be modified by the 
addition of new policies, removal of existing policies, or the updating of existing policies. In 
addition, this invention assumes that there are operations to remove domains and organizations 
when their agreements terminate. To add a new domain the client specifies the representation of 
the domain (including sub domains), policies, and a list of computing environments to 
provisioned for the domain. 

In Figure 11 the computing environment type is a web site. The BRDS 219 has a single root 
collector. There are BRLSs for network dispatcher license 1130, Linux servers (hardware and 
software) 1 132, and WebSphere license 1 134. When the new customer request 1 105 arrives, the 
new customer (NC) 1160 component of the BRDS 219 creates a new collector 1150 for it, 
(indicated by dotted lines) and a place holder for the customer's information until the PMRS 
1180 for the new collection is fully operational. The new customer request 1105 specifics the 
exact composite resource that will be instantiated. After creating the new collector the NC 1 160 
sends a Realize 1111 request to the PMRS 1 180 of the correct type. Realize is used by the BRDS 
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1 2 1 9 to get a set of possible plans for constructing a resource on the existing infrastructure when a 

2 new computing environment is requested. Realize is defined in detail in a separate invention 

3 entitled "Componentized Automatic Provisioning and Management of Computing Environments 

4 for Computing Utilities", having docket number YOR920030588PCT1. At the end of the 

5 Realize 1111 request the BRDS 219 has list of possible sets of resources that can be used to build 

6 the computing environment of the specified type. All of these options are returned to the NC 

7 1 160 object which instantiates a collector 1 150 to manage the new instance and request that the , 

8 collector 1150 to negotiate for the resources. The newly instantiated collector 1150 (BRDS) 

9 analyses the options against the available resources, and policies and negotiates for resources 

10 with the libraries BRLSs 1130, 1132, 1134. It then schedules the construction of the service at 

1 1 the requested time assuming all of the resources can be acquired within policy. (If the resources 

12 cannot be acquired the new customer 1105 request fails.) When the start time of the resource 

13 reservations arrives, the BRDS 219 obtains the resources and delivers the resource handles to NC 

14 1160, which sends a Build 1112 request to the PRS 1110. The Build request includes all 

15 required policies. One type of policy that is supplied is the acquisition policy, other required 

16 policies can be resource type specific and would be supplied at this time. It should be noted that a 

17 PMRS 1 180 is the logical association of a PRS 1 1 10 and an MRS 1 170. The details of the PMRS 

18 are described in an application entitled "Componentized Automatic Provisioning and 

19 Management of Computing Environments for Computing Utilities", having docket number 

20 YOR920030588PCT1. As a result of the Build 1 1 12 request the PRS 1 1 10 creates the MRS 1 170 : 

21 that will be associated with the computing environment. When this process completes the PRS 

22 1110 returns the identifier of the new MRS 1170 to the NC 1160, the NC 1160 updates the 

23 collector 1 1 50 with the identity of the root MRS 1 1 70 for the computing environment. 

24 Once the collector has the handle it is now associated with the newly instantiated service which it 

25 manages, as shown by the dashed line in Figure 1 1 . In an alternative embodiment, instead of 

26 having the NC send a build 1112 request to the PRS, the NC could send a DeliverResource 

27 request and the PMRS could realize that the first deliver resource request implies build. 

28 In an alternate embodiment, the function of the NC can be combined into the collector so that the 
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BRDS could instantiate the new collector when the new customer request arrives. The new 
collector would, as part of its initialization, go through all of the steps assigned to the NC. At the 
completion of those steps the NC would be done and the service operational. It is also possible 
that the hosted environment might not have the ability to satisfy the request, in which case the 
request for a new customer would be rejected. As indicated previously, this hosted environment 
has the ability to acquire and utilize resources that may be in some other hosted environment. 
Consequently, if this hosted environment does not have all of the resources required and it can 
obtain the resources from some other hosted environment with an acceptable set of conditions 
then the request for a new customer can be accepted instead of rejected. Resources obtained from 
another hosted environment would still be represented as a BRLS in this environment. When a 
hosted environment contains BRLSs that come from multiple hosted environments, the BRLSs 
are typed so that acquisition policies can be supplied to manage the use of external versus 
internal resources. 

Figure 12 shows an example of a distribution for provisioning purposes of a set of computing 
environments to Smith Fastener company 120. Smith Fastener had to decide which services they 
wanted to utilize within their organization and which part of the organization would hold (or 
subscribe to) each service. Smith also had to decide how to allocate resources to their domains 
and what the allocation and distribution policy would be for each domain. These provisioning 
decisions are orthogonal to which employees of Smith Fastener get to use the services that are 
provisioned. The guiding principle in producing the domain and computing environment tree is 
that a domain is only inserted where the is at least one computing environment to be provisioned, 
at least two sub domains to be established, or there is at least one computing environment and 
one sub domain. This principle prevents long strings of domains with nothing other than one sub 
domain. Smith Fastener Company 120 has decided that it wants to provision and control two 
applications at the corporate level application 1 and application 5. An accounting application, 
application 12 is to be used by corporate and each division Research 121,. Finance 122, Hardware 
123, and Marketing 124. The hardware division has decided that each product unit Bolts 127, 
Screws 128, and Hinges 129 should use their own instance of the same engineering design 
application 9. The Hinge unit 129, is also experimenting with two additional applications 
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1 application 1 0 and application 1 1 . Marketing has decided that each of its regions Region 1,125 

2 and Region 2, 126 should use their own instance of the same application, application 3, for 

3 managing marketing activities. Region 2, 126 of marketing is divided into two separate areas 

4 North 130 and South 131. Each of these regions uses their own instance of application 4 to 

5 collect information to feed into application 3 at the regional level The resource acquisition and 

6 distribution policies of Smith Fastener 1 20 assure that each of its major domains Research 121, 

7 Finance 122, Hardware 123, and Marketing 124 has the resources it needs to accomplish its 

8 tasks. In addition Marketing 124 and Finance 122 can have resources reallocated to them from 

9 other divisions, but not each other, during their peak end of quarter and end of year times. But 

10 they are not allowed to take so many resources as to disable the other divisions. 

1 1 When all of these decisions are mapped to a domain structure with computing environments it 

12 looks as follows: The root domain 120, has three applications AppOl 150, Appl2 151, and App 

13 05 152. The root domain 120 also has four sub domains Research 121, Finance 122, Hardware 

14 123, and Marketing 124. The sub domains of Research 121 and Finance 122 each use a single 

15 application. In this case they each have separate instances of the same application App 12 155 and 

16 Appl2 156. The present invention does not prohibit the sharing of a single instance of an 

17 application between domains from a user perspective. For example in Figure 12 AppOl could be 

1 8 used by both Research and Finance as well as the root domain. However, from a provisioning : 

19 (and management of provisioning) perspective each instance of an application has a single point 

20 of provisioning within the organization. Returning to the detailed description of the figure. The 

2 1 Hardware domain 1 23 has three sub domains and a single application associated with it. The sub 

22 domains of Hardware 123 are Bolts Unit 127, Screw Unit 128, and Hinge Unit 129. The 

23 application associated with hardware is App 12 157. The Marketing domain 124 has two 

24 applications and two sub domains associated with it. The application are 5 Appl2 153 and App03 

25 154 and the domains are Region 1, 125 and Region 2, 126. Each of those regions has its own 

26 instance of App03. Region 1, 125 uses App03 161 and Region 2, 126 uses App03 158. Region 2, 

27 126 is further divided into North 130 and South 131 and each sub domain of Region 2, 126 has 

28 its own copy of App04 North usees App 04, 1 63 and South uses App04, 1 62. 
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1 Figure 13 shows the collector hierarchy that results from the computing environment and domain 

2 structure of Figure 12. To produce the collector hierarchy we start at the root domain and insert a 

3 collector. Next we examine the root domain of the domain tree it to see if there is only one 

4 computing environment, if there is then we insert a PMRS for this computing environment and 

5 we are done. Otherwise, the root domain has more than one computing environment or sub 

6 domain. For each computing environment of the root domain we insert a collector and PMRS. 

7 The remaining sub domains are placed on the domain processing list. This algorithm is applied to 

8 each domain on the domain processing list until it is empty. Basically this algorithm is used 

9 breadth first search starting at the root domain of the domain and computing environment tree. 

10 When this algorithm is applied to Figure 12 the collector hierarchy show in Figure 13 results. 

1 1 PMRSs are typed, in Figure 1 3 the last two digits of the PMRS name are its type so PRMS01 is a 

12 PMRS which manages an instance of AppOl from Figure 12. Returning to a more detailed 

13 description of Figure 13, Collector SFC 1320 is inserted because Smith Fastener Company 120 in 

14 Figure 12 has more than one application. In Figure 12 Smith Fastener 120 has three applications. 

15 Therefore applications 1, 12, and 5 a collector PMRS structure was inserted. For application 1 

16 this is collector 1352 and PMRS01 1374, for application 12 this is collector 1351 and PMRS 1 2 

17 1373, and for application 5 this is collector 1350 and PMRS05 1371. Next we apply the 

1 8 algorithm to Smith's sub domains. We notice that tow of the sub domains in Figure 12 Research 

19 121 and Finance 122 only have a single application, application 12. Consequently for each of 

20 these sub domains a collector and PMRS structure is inserted into the hierarchy. For Research 

21 121 in Figure 12 collector 1321 and PRMS12 1370 in inserted and for Finance 122 in Figure 12 

22 collector 1322 and PMRS 12 1372 is inserted. In Figure 12 Hardware 123 and marketing 124 

23 have multiple sub domains and application. Next we apply the algorithm to the sub domains of 

24 Hardware and Marketing. For hardware collector 1 323 is inserted and for Marketing collector 

25 1324 is inserted. In Figure 12 hardware has three sub domains Bolts 127, Screws 128 and 

26 Hinges 129 and one application, application 12 associated with it. So a PMRS collector hierarchy 

27 is added to collector 1323 for the application. In Figure 12 two of Hardware sub domains have 

28 only a single application associated with them Bolts 127 and screws 128. In Figure 13 a collector 

29 PMRS structure is associated with each of these sub domain. For Bolts it is collector 1327 and 

30 PMRS09 1376 and for screws it is collector 1328 and PMRS09 1377. In Figure 12, the third unit 
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1 of Hardware 123, Hinges 129 has three applications associated with it so in Figure 13 collector 

2 1 329 is inserted. In Figure 1 2 marketing has two applications and two sub domains. These 

3 application are Appl2 153 and App03 154 and the sub domains are Region 1, 125 and Region 2, 

4 126. In Figure 13 we have inserted a collector PMRS hierarchy for each of the applications. 

5 These are collector 1354 and PMRS 12 1378 for application 12 and collector 1355 and PMRS03 • 

6 1379 for application 3. In Figure 12 one sub domain, North 130 has only a single application 

7 associated with it, App04 163. In Figure 13 a collector PMRS hierarchy is inserted. This is 

8 collector 1330 and PMRS 04 1384. In Figure 12 the other sub domain of marketing, Region 2 

9 1 26 has an application and two sub domains. Consequently in Figure 1 3 a collector is inserted 

10 collector 1326. Returning to Hinges 129 in Figure 12, it has three applications associated with it 

1 1 Appl 1 164, App09 165, and Appl 0 166. In Figure 13 A collector PMRS hierarchy is inserted for 

1 2 each of these applications . These are collector 1 3 5 6 and PMRS 11 1 3 8 1 for application 1 1 ; 

13 collector 1357 and PMRS09 1382 for application 9; and collector 1358 and PMRS10 1383 for 

14 application 10. Finally we examine Region 2 126 in Figure 12. It has one application App03 158 

1 5 and two sub domains. In Figure 1 2 each of its sub domains North 1 30 and south 1 3 1 has an 

16 instance of application 4. In Figure 1 3 we insert a PMRS collector hierarchy for application 3, 

1 7 collector 1 359 and PMRS03 1 386. For the sub domains we also insert a collector PMRS 

.« 

1 8 hierarchy for each one because they only have one application. This is coPector 1 330 and 

19 PMRS04 1384 for North and collector 1331 and PMRS 04 1385 for South'. This completes 

20 construction of the collector computing environment hierarchy which will be used to provide 

2 1 hierarchical management of resources by this invention. 

22 The collector tree is designed to allow maximum flexibility in provisioning services to a 

23 customer. It allows private BRLSs to be associated with each collector in the tree. Each collector 

24 can have its own acquisition and distribution policies. This allows the company, in this case 

25 Smith Fastener, the ability to more finely control the distribution of its computing resources. If 

26 all acquisition and distribution policies are identical and all resources are held in Public BRLSs 

27 then this tree is unnecessary and a flat tree would be sufficient. By using a hierarchy, a finer level 

28 of control is obtained over the distribution and use of resources within an organization. 
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1 The present invention is useful with the inventions described in the above cross-referenced 

2 applications entitled "Componentized Automatic Provisioning and Management of Computing 

3 Environments for Computing Utilities", and entitled "Apparatus for Arbitration in a Computing 

4 Utility System". One example of a use is to combine the inventions to provide on-demand 

5 services to a set of customers. 

6 Variations described for the present invention can be realized in any combination desirable for 

7 each particular application. Thus particular limitations, and/or embodiment enhancements 

8 described herein, which may have particular advantages to a particular application need not be 

9 used for all applications. Also, not all limitations need be implemented in methods, systems 

10 and/or apparatus including one or more concepts of the present invention. 

1 1 The present invention can be realized in hardware, software, or a combination of hardware and 

12 software. A visualization tool according to the present invention can be realized in a centralized 

13 fashion in one computer system, or in a distributed fashion where different elements are spread 

14 across several interconnected computer systems. Any kind of computer system - or other 

15 apparatus adapted for carrying out the methods and/or functions described herein - is suitable. A 

1 6 typical combination of hardware and software could be a general purpose computer system with 

1 7 a computer program that, when being loaded and executed, controls the computer system such 

1 8 that it carries out the methods described herein. The present invention can also be embedded in a 

1 9 computer program product, which comprises all the features enabling the implementation of the 

20 methods described herein, and which - when loaded in a computer system - is able to carry out 

21 these methods. 

22 Computer program means or computer program in the present context include any expression, in 

23 any language, code or notation, of a set of instructions intended to cause a system having an 

24 information processing capability to perform a particular function either directly or after 

25 conversion to another language, code or notation, and/or reproduction in a different material 

26 form. 
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Thus the invention includes an article of manufacture which comprises a computer usable 
medium having computer readable program code means embodied therein for causing a function 
described above. The computer readable program code means in the article of manufacture 
comprises computer readable program code means for causing a compute) to effect the steps of a 
method of this invention. Similarly, the present invention may be implemented as a computer 
program product comprising a computer usable medium having computer readable program code 
means embodied therein for causing a function described above. The computer readable program 
code means in the computer program product comprising computer readable program code 
means for causing a computer to effect one or more functions of this invention. Furthermore, the 
present invention may be implemented as a program storage device readable by machine, 
tangibly embodying a program of instructions executable by the machine to perform method 
steps for causing one or more functions of this invention. 

It is noted that the foregoing has outlined some of the more pertinent objects and embodiments of 
the present invention. This invention may be used for many applications. Thus, although the 
description is made for particular arrangements and methods, the intent and concept of the 
invention is suitable and applicable to other arrangements and applications. It will be clear to 

♦ 

those skilled in the art that modifications to the disclosed embodiments can be effected without 
departing from the spirit and scope of the invention. The described embodiments ought to be 
construed to be merely illustrative of some of the more prominent features and applications of the 
invention. Other beneficial results can be realized by applying the disclosed invention in a 
different manner or modifying the invention in ways known to those familiar with the art. 
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